Prediction of Martensite Start Temperature in Steels Using Cross Validation

Introduction

Cross-validation Overview

• A statistical technique for evaluating the performance and generalizability of machine learning models.

• Divides dataset into training and validation subsets.

• Ensures model training on one subset and validation on another.

Advantages:

• Provides more reliable estimates of model performance.

• Reduces bias compared to a single train-test split.

• Improves model generalizability by leveraging different training and validation data.

Methods

• K-Fold Cross-Validation: Dataset is split into k folds, model is trained on k-1 folds and validated on the remaining fold, repeated k times Kohavi (1995).

• Leave-One-Out Cross-Validation (LOOCV): A special case of K-Fold where k equals the number of observations, each sample serves as the validation set once.

• Nested Cross-Validation: Used for model selection and hyperparameter tuning, an outer loop for validation and an inner loop for training and hyperparameter optimization.

Model Measures of Error (MOE)

• Definition: Measures of Error (MOE) quantify the difference between predicted values and actual outcomes, helping assess model performance.

Models

• First Model (Linear Regression):

Ms = 769.41 -286.71 C -16.42 Mn -14.04 Ni - 13.89 Si - 10.13Cr -41.45C:Mn - 8.36 C:Ni

Models

• Second Model (Log-Linear):

log(Ms) = -6.69 - 0.51C - 0.03 Mn - 0.03 Ni - 0.03 Si - 0.02Cr - 0.07 C:Mn - 0.01C:Ni

Model Performance Summary

Summary

Data Exploration and Visualization

In our study, we analyzed a dataset from Wentzien et al. (2024) Martensite dataset focuses on predicting the Martensite Start Temperature (Ms) in steel alloys based on their chemical compositions.

• Martensite start temperature (Ms) is target variable.
• “C”,“Mn”,“Si”,“Cr”,“Ni” are Predictor variables

Correlation_Matrix

Modeling and Results

Linear Regression Model

Linear regression is a fundamental statistical technique that establishes a relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data.

In our dataset, which focuses on predicting the Martensite Start Temperature (Ms) of steel based on its chemical composition (C, Mn, Si, Cr, Ni), linear regression allows us to quantify how changes in these elements influence Ms.

M_s = \beta_0 +\beta_1 C +\beta_2 Mn + \beta_3 Si + \beta_4 Cr + \beta_5 Ni

M_s = 746.99 - 254.85 C - 24.24 Mn - 13.28 Si - 7.8 Cr - 14.64 Ni

Statistics

Residual standard error: 54.28 on 1230 degrees of freedom

Multiple R-squared: 0.7433,

Adjusted R-squared: 0.7422

F-statistic: 712.2 on 5 and 1230 DF,

p-value: < 2.2e-16

Cross Validation Results for Linear Regression

The results of this analysis reveal that the models tested with 5-Fold Cross-Validation (5-Fold CV) and Leave-One-Out Cross-Validation (LOOCV) demonstrate impressive predictive accuracy. These two models consistently outshine the Nested CV model, indicating that they are more dependable for making predictions from the dataset.

Support Vector Machines (SVM) for Regression (SVR)

Support Vector Machines (SVM) are algorithms that model data by finding optimal boundaries, handling nonlinear patterns using kernels. Using our dataset, Support Vector Machines (SVM) with a radial kernel help predict the martensite start temperature (Ms) based on chemical elements like C, Mn, Ni, Si, and Cr. SVM works by finding the best way to capture the relationship between these variables, effectively handling complex patterns for accurate Ms predictions.

Model Comparision Results

In this study, we compared the performance of Linear Regression and Support Vector Machine (SVM) models in predicting the martensite start temperature (Ms). Using 5-fold cross-validation, we found that SVM outperformed Linear Regression, achieving a lower MAE (~21 vs ~33), a higher R² (~0.9 vs ~0.55), and a lower RMSE (~25 vs ~48), highlighting its superior accuracy and reliability in making predictions.

Model Comparision Plot

Model Comparision Results Plot

Conclusion: Overview

Evaluation of Two Models:

Linear Regression Model

Support Vector Machine Model

Cross-validation Methods Used:

k-fold Cross-validation

Leave-one-out Cross-validation (LOOCV)

Nested Cross-validation

Conclusion: Key Findings

• Mean Absolute Error (MAE): SVM performed much better, with a lower MAE (~21) compared to Linear Regression (~33), meaning SVM’s predictions were closer to the actual values.

• R-squared (R²): SVM showed a significantly higher R² (~0.9) than Linear Regression (~0.55), indicating that SVM explained 90% of the data’s variability, while Linear Regression only accounted for about 55%.

• Root Mean Squared Error (RMSE): SVM had a much lower RMSE (~25) compared to Linear Regression (~48), which reflects its greater accuracy and fewer large prediction errors.

• Overall Performance: Across all key metrics, SVM outperformed Linear Regression, proving to be a more accurate and reliable model for predicting martensite start temperature (Ms).